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(57) Abstract 

An optical in situ information system includes a step of generating a pair of pictures of a synthetic object, taking into account 
parameters of the viewing optical system. Next, the left and right eye (stereoscopic) images are appropriately combined with the respective 
real object as viewed through two separate oculars. The synthetic image can be made to overlay the image of the real object over a full 
three-dimensional space. 
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OPTICAL IN SITU INFORMATION SYSTEM 

Background of the Invention 

1 . Field of the Invention 

The present invention relates to an optical in situ 
5 information system- In particular, the present invention 
relates to a system that presents a wide variety of 
information (synthetic objects) in situ with a field of 
view of a view looking through an optical system, such as 
a microscope or a different type of viewing device with 
10 arbitrary magnif ication. 

2 . Description of the Related Art 

There are several types of conventional systems that 
enable a viewer to view synthetically produced pictures. 
One such type of conventional system is a virtual reality 

15 system. In such systems/ stereo pairs of a scene are 
totally synthesized (generated) with computers, and then 
viewed through a viewer with a binocular viewing device. 
In virtual reality systems, no attempt is made to overlay 
the synthetic view onto a view of real, bright objects. 

2 0 In fact, typically virtual reality systems are used in a 
dark environment, so that synthetic imaiges of high 
brightness are not needed. 

Another type of conventional system is a head-mounted 
display, as used by pilots and the like. In such 

25 systems, the objects in the field of view of the pilot 
are so distant that they do not produce a perceptible 
stereo-effect. Consequently, the field of view is 
essentially a two-dimensional plane without any depth 
perception. As a result, the synthetic pictures 

30 superimposed over the pilot's view do not have any 
requirements of depth perception. 



BNSDOCID: <WO 9813716A1J_> 



WO 98/13716 



2 



PCI7US97/17264 



10 



Still another type of conventional system is a 
display built into surgical microscopes. such systems 
are described, for example, in U.S. Patent No. 4,202,037 
issued to E.M. Glaser, and in U.S. Patent No. 4,722,056' 
issued to Roberts et al. Glaser describes a system for 
simultaneous viewing (without aligned overlay) of 
computer generated information in the field of view of a 
surgical microscope. Roberts describes a method of 
producing aligned overlays. This method is limited to 
overlays of two-dimensional pictures correctly aligned 
only in the focal plane of the microscope, m principle 
Roberts' method is similar to the head mounted displays 
used by pilots, and only provides a monoscopic view. 

Summary of the Invention 

L5 one object of the present invention is to provide a 

sufficient light capability to view a synthetic image 
against a bright real image. 

Another object of the present invention is to provide 
a stereoscopic view of a combined image of a synthetic 

0 image with a real image. 

Still another object of the present invention is to 
provide a stereoscopic, combined image of a three- 
dimensional synthetic image with a three-dimensional real 
image . 

5 The present invention relates to a method by which 

computer generated images of synthetic three dimensional 
objects can be correctly overlayed and aligned over the 
images of real, three-dimensional, objects in the field of 
view of a viewer. The embodiments described herein also 
0 include many novel features and techniques which will 
improve brightness, contrast, resolution, ease of 
aduustment, and operation over the conventional systems 
for aligned and unaligned overlays of two dimensional as 
well as three dimensional objects. 

The method includes a step of generating a synthetic 

P* ir °f Pictures of a three-dimensional 
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synthetic object which correspond to the synthetically 
produced three-dimensional picture. The method also 
includes a step of respectively combining the synthetic 
stereoscopic pair of pictures with a real stereoscopic 
5 pair of pictures which correspond to respective images 
the real object, the real stereoscopic pair of pictures 
being respectively obtained from two oculars of a 
microscope. By this method, the synthetically produced 
picture is made to overlay the images of the real object 
10 over a three-dimensional space and not just over a single 
focal plane. 

Brief Description of the Drawings 

These and other objects and advantages of the 
invention will become more fully apparent from the 
15 following detailed description when read in conjunction 
with the accompanying drawings with like reference 
numerals indicating corresponding parts throughout, and 
wherein: 

Figure 1 shows a conventional viewing system using 

20 a microscope; 

Figure 2 shows a conventional stereoscopic viewing 
system using a microscope in order to achieve a 

stereoscopic image ; 

Figure 3 shows a system for generating an overlay of 
25 images in a single focal plane according to a first 
embodiment of the invention; 

Figure 4 shows a system for generating a stereoscopic 
image according to either a second or third embodiment of 

the invention; and 
30 Figures 5A and 5B shows a difference in the mapping 

of LCD pixels for a synthetic image between the second 
and third embodiments, with respect to a reference plane 
and a different plane. 
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i„fo The / reSent inVenti °" relat " *o •» optical in situ 
° 1M SyStem ' WMCh PreSe " tS a variety of 

5 ■ < SynthetiC ° b ^ot S > in situ with the field of 

view of a Viewer looking through an optical system such 
as a horoscope or other type of viewing device with 
arbitrary magnification. The term -microscope" used 
throughout will be interpreted to include microscopes! 

10 "IT *" ^ ° PtlCal SyStMS attached t o 

goggles. T „e te rm -synthetics objects" is used 

throughout to emphasize that the information displayed by 
the present invention is synthesized with computers and 
electrons, and it may represent two-dimensional as well 
as three-dimensional objects. 

15 obiect ,*■ USUal ° Perati ° n ° f " 1«-oop.. light from an 
of th. " SOUr ° e °' ln£ ° rmatl o n ) « the object space 

Ixits th r °r° Pe niCro -ope objective and 

of the ob "I °* mi ° r ° S °°** '« viewing the image 
of the object, In an extension of this concept, right is 
directed towards an object loo by a light source 105 A 
portion of the light directed towards the object 100 
reflects of, of the object no and enters a microscope 



25 



30 



35 



The light that enters the microscope no is split by 
a beam splitter 120, after passing through oculars 130. 
One portxon of the output from the beam splitter 120 is 
directed to an imaging device 140, such as an electronic 
camera, as shown in Figure 1. The other portion of light 
output from the beam splitter 120 is directed to an eye 

for °.Vh Vi6Wer * ^ 6leCtronic —a 140 may be used 
for either viewing the image on a TV monitor or for 
electronxc analysis of the image. 

For stereoscopic viewing of the object, the object 
is imaged from two different angles, and viewed through 

Flir^^ t0 f ° rra 3 Stere ° SC °P ic ***** of the object. 
210 aY 3 C ° nVentional — ot a first microscope 

210 and a second microscope 220, which together provide 
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stereo signals to a TV monitor 230, in order to obtain a 
stereoscopic image of the object. 

A first embodiment of the invention corresponds to 
a system for simultaneous viewing of a real object and a 
synthetic object. In the first embodiment, light from 
two separate sources is combined by an optical combiner 
310, as shown in Figure 3. The first source of light is 
a synthetic image, created by a computer 34 0 that outputs 
a video signal to an LCD 350, where the LCD 350 is 
illuminated by a light source 3 60. The output of the 
LCD 350 is combined with light received from the object 
100 via the objective lenses 320. The combined light 
output from the optical combiner 310 passes through an 
ocular 3 30, which focuses the light so that both objects 
15 can be viewed simultaneously in the field of view of a 
viewer 160. In Figure 3, the ocular 330 and the 
objective 32 0 are both shown as having two separate lens 
spaced apart from each other, but one of ordinary skill 
in the art will recognize that many different types of 
20 oculars and objectives may be utilized while keeping 
within the scope of the invention. 

In the first embodiment, one of the images being 
viewed corresponds to an image from a real object, while 
the other image being viewed corresponds to a 
25 synthetically produced picture or text input through 
electronics, such as created using a liquid crystal 
display (LCD) . In the first embodiment as described 
above, simultaneous viewing of a real object and a 
synthetic object is thereby achieved. By the use of the 
30 LCD 350 with the appropriate amount of light applied to 
it by the light source 360, one is able to clearly view 
a synthetic image overlayed with a bright, real image 

from the object 100. 

In order to achieve stereoscopic viewing, a second 
35 embodiment of the invention uses two separate systems - 
one for the left eye and one for the right eye, with the 
two images paraf ocalized to produce an overlay of images 
onto a single focal plane for viewing by a viewer. 
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Figure 4 shows a system according to a second embodiment 
of the invention, in which stereoscopic viewing is 
achieved. 

The second embodiment includes a first computer 410A 
5 which outputs electrical signals (for example, a video 
signals) responding to a "left eye synthetic image" to 
LCD and optics Unit 420A. LCD and Optics Unit 420A 
includes a matrix array of LCD pixels, as well as optics 
for outputting the light from the pixel array at a 
10 particular magnification and brightness. LCD and Optics 
Unit 420A receives light from Light Source 430A, via a 
fiber optic cable (or cables, or the like, and outputs an 
illuminated synthetic image to beam combiner 440A. 

Illuminators 450A and 450B provide light in a 
15 direction of a real object 490. Illuminators may be 
configured as lamps that output light across a wide 
optxcal spectrum. Some of the light incident on the real 
object 490 is reflected in a direction towards Objectives 
460A and 460B. Objective 460A is an objective lens for 
20 focusing light to be eventually directed to a left eye of 
a vxewer, and objective 460B is an objective lens for 
focusing light to be eventually directed to a right eye 
of a viewer* 

Light output from objective 460A is directed to beam 
15 combiner 440A, where the "real image" from the real 
object is combined with the "synthetic image" received 
from the LCD and Optics Unit 420A. The combined 
synthetic and real" light passes through ocular lenses 
470A, and is focused onto a left eye 495A of the viewer 
0 Similarly, reflected light is also received by 

objective 460B, and is directed to beam combiner 440B 
Second computer 410B creates a video signal for a "right 
eye synthetic image", which is sent to LCD and Optics 
Umt 420B. LCD and Optics Unit 420B also receives light 
o from Light Source 4 3 OB, via a fiber optic connection or 
the like. The illuminated synthetic image output from 

LCD and Optics Unit 420B is sent i-^ 

° iS seirc to beam combiner 4 4 OB 
where it is combined wii-h *-v.~ ' 

Q Wxth the real image corresponding to 
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light received from the real object. In the second 
embodiment, the LCDs of LCD and Optics Units 4 2 OA and 
4 2 OB correspond to a matrix of pixels, arranged in a two- 
dimensional grid. The combined light from beam combiner 
5 440B passes through ocular lenses 470B, and is focused 
onto a right eye 4 95B of the viewer. 

By the system according to the second embodiment, the 
viewer can see a stereoscopic viewing of a real image 
superimposed with a synthetic image. Such- a 

10 configuration is useful in many medical procedures. For 
example, a CATSCAN or a selected portion thereof, as a 
synthetic image, can be superimposed onto a real image 
corresponding to an actual patient situated in a viewing 
region of the system according to the second embodiment. 

15 In the second embodiment, a calibration is performed 

so that the synthetic image can be properly imaged onto 
one or more pixels of the LCD array in the LCD and Optics 
Units. That calibration may be performed in a number of 
ways, such as providing a test image at a particular 

2 0 distance and angle with respect to the viewing system, 
and then determining where that test image appears on an 
X-Y array of LCDs. Based on this information, which is 
taken at a variety of angles of objects with respect to 
the viewing system, a synthetic image can be properly 

2 5 superimposed onto a real image by knowledge of the actual 

position of the real object with respect to the viewing 
system as well as knowledge of which pixel (or pixels) of 
the LCD array would be illuminated by the object at that 
position. 

3 0 In the second embodiment, a reference plane 

corresponds to the plane of the LCD array of pixels, and 
the synthetic image is determined based only on where it 
would appear on the reference plane, and no other plane. 
Thus, the combined image of the real image and the 
3 5 synthetic image is exactly valid for a real object on the 
reference plane, and may not provide an acceptable 
overlay of the synthetic image onto the real image when 
the real object is not on the reference plane. The 
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second embodiment does not require complex calculations 
however, and may be acceptable for some situations where 
an exact, overlay is not necessary. However, where a 
precise overly is required, such as in complex surgery 
for example, a different scheme is desired. 

in that regard, a third embodiment of the invention 
produces a three-dimensional overlay of a synthetically 
produced picture over a three-dimensional image of the 
real ob 3 ect. The third embodiment has a structure 
sxmilar to the second embodiment of Figure 4, but the 
computer in the third embodiment provides the means for 
creating a three-dimensional synthetic image that can be 
superimposed onto the three dimensional real image, and 
which will appear realistic when the real object is 
viewed from any angle and/or distance. 

First, a synthetic stereoscopic (left and right eye) 
pair of pictures of the synthetic object is calculated bv 
the computers 410A and 410B, and then displayed on the 
LCDs 420A and 420B, taking into account the parameters of 
20 the viewing optical system. These parameters may be 
calculated from pre-calibrated correspondence between 
points in the three-dimensional object space and points 
xn a plane of the LCDs. This method is illustrated by 
Figures 5A and 5B, which show a correspondence between 
the real object space and the LCD space. Referring now 
to Figure 5B, as the real object moves from a reference 
Plane 510 to a different plane 520 (i.e., at a different 
distance with respect to the microscope), a point A- on 
the different plane 520 maps to a different pixel 
position B on the reference plane 5io. 

In the second embodiment, the synthetic object is 
considered to be a two dimensional object in the focal 
Plane of the microscope. Therefore, in the calibration 
process of the second embodiment, a correspondence 
between the points in the focal plane of the microscope 
and points in the LCD plane is determined. m case the 
actual synthetic object is a three dimensional object 
either a cross-section of this three dimensional object 



25 



30 



35 
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or 



a collapsed projection of this three dimensional 
object is displayed in the LCDs to produce an overlay in 
the focal plane of the microscope. 

In the second embodiment, the point A • on the 
5 different plane 520 would be collapsed to the point A on 
the reference plane 510, since the synthetic image is 
only overlayed in a two-dimensional manner onto the real 
image. Figure 5A shows the locations of a pixel A and an 
adjacent pixel B on an LCD plane, where the LCD plane 
10 includes a plurality of LCDs (not shown) positioned in a 
matrix arrangement. The intersection of the dashed and 
dotted lines in Figure 5B corresponds to a location of 
the viewing system with respect to the reference plane 
510 and the different plane 520. 
15 if the real object is on the reference plane, then 

the pixel illuminated by the system according to the 
second embodiment would correspond to the pixel 
illuminated by the third embodiment. However, if the 
object is moved a particular distance away from the 
20 reference plane (i.e., either closer to or farther away 
from the microscope) , then the pixel illuminated by the 
second embodiment would be different from the pixel 
illuminated by the third embodiment, as shown in Figure 
5B. In essence, the LCD array is configured to make the 
25 synthetic image appear at a particular distance with 
respect to the microscope, with that particular distance 
corresponding to the reference plane. 

In the third embodiment, the means for determining 
the mapping of a three-dimensional real space onto a two- 
3 0 dimensional LCD pixel array is preferably performed 
beforehand by calibration, based on real images received 
from a real test object, and by taking into account the 
distance and angle of the viewing system with respect to 
the real test object. At the same time, the computers 
3 5 410A and 410B determine, for the X-Y grid of LCD pixels 
on the LCDs 4 2 OA and 4 2 0B, where the real object would 
appear (i.e., which pixel in the LCD array should be 
illuminated) in order to create an appropriate three- 
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dimensional synthetic image to be overlaid onto the 
three-dimensional real image from the ob;ject 490 

The system according to the third embodiment takes 

col : B seT\ h SySte " aCCOrdin9 SeC ° nd — -i—t 

collapses the synthetic image onto a single (reference, 

X V T J" , W ° rdS ' ln th£ third — diment, the 

X,Y 2 object space corresponds to a particular pixel 
number (x,., y L coordinate, on the left VC D array, and a 
10 particular pixel number fx y, „, (fc • w 

mL 0 ■ -a .. . b) the ri9ht LCD array. 

The third embodiment provides a three dimensional mapping 
onto the useful space, „ hi i e the secon<J 

ZZ :: ' dimenSi °"" - synthetic image 

onto the useful space. In the third enibodi J, 

Ld'oo:" ^ reSP6Ct t0 the ^ 

"If! 1 " !:" S "° B ^ bS to make 

synchetf aPPSar (that 1S USed *" «~f t-. 

re" obH C :T e l ^ COMe fr ° m 3 SUita " e i» «- 

20 at anoth. T ^ ° f the ° b ^«- °r 

at another plane, which in the third embodiment is 

operator-adjustable, . The system thus provides a more 
comfortable viewing of the overlay of the synthetic image 
with the real image. 

2 S also "trusted ^ < ^ ~<*«- -» 

„„~- , ad:l " sted s ° «*« light appears to be at a 
Particular distance away, and not necessarily at an 
"infinite" aistance away. Tha t way, a realistic three" 
dimensiona viewing of an object together with a three- 
dimensional synthetic image may be achieved, where the 

30 viewer actually "feels" th» 

sh e .« „ distance and angle that he or 

she is with respect to the object. 

in the third embodiment, the synthetic image as 
viewed by human eves i c- +.„ 

t-^i k - ^ ° overla y the image of the 

real object over the useful three-dimensional space and 

The means for generating synthetic stereoscopic pairs 
of pictures can be performed via li guid crystal displays 
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(LCDs) , or other similar ways. Note that cathode ray 
tubes (CRTs) generally do not provide the amount of light 
and resolution necessary to superimpose a synthetic image 
onto a real, bright image, and so that is why LCDs are 
5 preferred in each of the embodiments described herein. 
In each of the embodiments, fiber optics are used in 
order to provide a light conduit between the light source 
and the liquid crystal displays, thereby allowing a 
strong source of light to be placed at a distance. The 

10 present invention may also include imaging means for 
imaging the distal tip of the optical fiber at the 
entrance pupil of the viewing ocular system, to allow an 
efficient coupling of light to the eye. The present 
invention may include adjusting means for adjusting the 

15 LCD illumination intensity, so as to equalize the left 
and right LCD brightness level. The light adjusting 
means may be implemented within the light sources that 
provide light for the LCDs. By proper light adjusting, 
a common brightness level can be made compatible with the 

20 illuminated field of view of the microscope. 

In the second embodiment, the left and right 
synthetic images are paraf ocalized to a common reference 
plane, while in the third embodiment, the left and right 
synthetic images are paraf ocalized to an optimum 

25 reference plane in the real object space. The 
magnification of the left and right synthetic images can 
be adjusted to match each other, via an adjusting means 
for left and for the right images. The magnification of 
the LCD display can also be adjusted via the optics 

3 0 provided in the LCD and Optics units, in order to adjust 
the ocular field of view to be covered by the display 
images * 

The present invention allows for calculating 
stereoscopic views from a data base defining: i) the 
3 5 synthetic three dimensional object, and ii) pre- 
calibrated correspondence between the three-dimensional 
space and the points in the plane of the LCDs. The data 
base is preferably stored in a memory that is accessible 
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by a computer. The computer preferably has embedded 
software for performing the above-mentioned calculation 
of the stereoscopic views. A transformation matrix may 
be utxHzed in order to determine which pixel in an X-Y 
array of pixels would be illuminated based on a 
particular angle and distance of the viewing system with 
respect to the real object. 

The invention allows for the ability to adjust the 
contrast of the pictures produced by the LCDs (i e 
either lightening them or darkening them, as needed)" 
based on the brightness of the real image from the real 
object. 

The invention also may be configured with X, Y and 
theta adjusting means for making x, y and theta 
adjustments so as to allow for an adjustable overlay of 
the left and right eye views of the synthetic images 
within the field of view of the ocular (i.e., zoom in or 
zoom out the size of the synthetic image overlayed on the 
real image) . 

The invention also may include measuring means for 
measuring the parameters of the system needed to overlay 
the left and right images. 

Also, the parameters of the optical system containing 
variable lenses may be dynamically measured in order to 
dynamically maintain the overlay of the synthetic images 
in the field of view. 

Still further, the present invention may also include 
dynamic measuring means for dynamically measuring the 
parameters of the optical system to dynamically maintain 

30 the overlay of the synthetic images in the field of view. 

While embodiments have been described herein 
modification of the described embodiments may become 
apparent to those of ordinary skill in the art, following 
the teachings of the invention, without departing from 

35 the scope of the invention. For example, the present 
invention has many applications, including: image guided 
surgery, image guided therapy, ima ged guided diagnostics 
telesurgery, robotic surgery, and inspection system to 
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compare real objects with their three dimensional 
reference profiles. 
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What Is ClaimPd Ts : 

1. A method of stereoscopic viewing of a real 
object and a synthetic object, the method comprising the 
steps of: 

a) illuminating the real object; 

b) receiving light reflected off of the real 
object; 

c) creating a left eye and right eye stereoscopic 
synthetic image ; 

d) combining the left eye stereoscopic synthetic 
image with the received light from the real object as a 
first combined image; 

e) combining the right eye stereoscopic synthetic 
image object with the received light from the real object 
as a second combined image; and 

f) providing the first combined image to a first 
eye of a viewer and providing the second combined image 
to a second eye of the viewer. 

2. The method according to claim l, wherein the 
left eye and right eye stereoscopic synthetic images are 
created based on a computer output of video data, and 

wherein the video data is converted to the left and 
right eye stereoscopic synthetic image by providing the 
video data to a first and a second liquid crystal display 
(LCD) , respectively. 

3. The method according to claim 2, wherein a light 
source provides light for the first and second LCDs, 
thereby providing a particular intensity for the left eye 
and right eye stereoscopic synthetic images. 

4. The method according to claim 3, wherein the 
particular intensity is adjustable. 

5. The method according to claim 2, wherein the 
left eye and right eye stereoscopic synthetic images are 
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mapped onto a two dimensional space corresponding to a 
particular pixel of a matrix of pixels of the first and 
second LCDs , respectively, 

6. The method according to claim 5, wherein the 
particular pixel of the matrix of pixels of the first and 
second LCDs is actuated to create the left eye and right 
eye stereoscopic synthetic images, irrespective of an 
actual angle of the real object with respect to the 
viewer. 

7. The method according to claim 5, wherein the 
particular pixel of the matrix of pixels of the first and 
second LCDs is actuated to create the left eye and right 
eye stereoscopic synthetic images, based in part on an 
actual angle of the real object with respect to the 
viewer . 

8. An apparatus for stereoscopic viewing of a real 
object and a synthetic object, comprising: 

an illuminator for illuminating a real object; 

first and second objective lenses for receiving light 
reflected off of the real object; 

a computer for creating a left eye and right eye 
stereoscopic synthetic image and for respectively 
outputting a first and second video signal as a result 
thereof ; 

a first and a second liquid crystal display (LCD) for 
respectively receiving the first and second video signals 
and for providing a first and second synthetic image; 

a first combiner for combining the first synthetic 
image with the received light from the first objective 
lens as a first combined signal; 

a second combiner for combining the second synthetic 
image with the received light from the second objective 
lens as a second combined signal; and 

first and second oculars for respectively receiving 
the first and second combined signals and for 
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respectively directing the first ^ 

_ rxrst and second combined 

signals to a left eve anri = 

. . y and 3 rignt e V e of a viewer, so as 

to achxeve a stereoscopic viewing. 

first 'and Th s e acc ~ di "* to claim 8, wherein the 

"* Se *° LCDS "«* -^ise a two-dimensional 
array of pixels. 

10. The apparatus according to claim 8, further 
comprising: tner 

a light source for outputting light; an d 
at least one fiber optic line for providing the light 
output from the li ght source to 9 light 

LCDs Iirst ar »d second 

the Sr el , n ° PtlC 11,16 Pr ° VldeS «* »** to 

the LC Ds for creation of the left eye end right eye 

stereoscopic images. y 

11. The apparatus according to claim 10, further 
compr.s.ng light adjusting means for varying an mtens^y 

In Inten 1 t ^ "* t *° - «r£ 

svnthet ^ "* ri « ht stereoscopic 

synthetic images. 

12. The apparatus according to claim 8. wherein the 

maolr T ^ Stereosc °P- synthetic images are 

»a P ped onto a two dimensional space corresponding to a 
particular p lX el of a matrix of pixels of the f J st ^ 
second LCDs, respectively, 

Bart "', The "PP^atus according to claim 12, „ he rein the 
particular prxel of the matrix of pixels of the first and 

"nt^c 1 s actuated to create the ieft 

eye stereoscopic synthetic images, irrespective of an 
actual angle of the real object with respect to the 
viewer. tn€ 
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14. The apparatus according to claim 12, wherein the 
particular pixel of the matrix of pixels of the first and 
second LCDs is actuated to create the left eye and right 
eye stereoscopic synthetic images, based in part on an 
actual angle of the real object with respect to the 
viewer . 
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